’* •> : / 


4 T* 


AUGUST, 1967 


REPORT ESL-R-317 
M.I.T. PROJECT DSR 76265 

NASX Research Grant NGR-22-009(124) 




THE MATRIX MINIMUM PRINCIPLE 

GPO PRICE $ 

V 

CFSTI PRICE(S) ^ 

Hard copy (HC) 3JL 

Michael Atbans Microfiche (MF) 

ff 653 July 65 


N67-39574 

(ACCESSION NUMBER) 


>; C7- J \ u' JL £ 1=1 

j c/^nUr 

(NASA CR OR TMX OR AD NUMBER) 



Electronic Systems Laboratory 


MASSACHUSETTS INSTITUTE OF TECHNOLOGY, CAMBRIDGE, MASSACHUSETTS 02139 


Department of Electrical Engineering 


August 1 967 


Report ESL-R-317 


Copy » «- 


THE MATRIX MINIMUM PRINCIPLE 

by 

Michael Athans 


This research was carried out at the M.I.T. Electronic Systems 
Laboratory with support extended by the National Aeronautics and 
Space Administration under research grant NGR-22 -009(124), 

DSR Project No. 76265. / 


^ El ectronic Systems Laboratory 
Department of Electrical Engineering 
| Massachusetts Institute of Technology 
Cambridge, Massachusetts ^02139 


PRECEDING 


page blank not filmed. 


ABSTRACT 

The purpose of this report is to provide an alternate statement of the 
Pontryagin maximum principle as applied to systems which are most 
conveniently and naturally described by matrix, rather than vector, 
differential or difference equations. The use of gradient matrices 
facilitates the manipulation of the resultant equations. The theory is 
applied to the solution of a simple optimization problem. 
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I. INTRODUCTION 


The purpose of this report is to provide (with no proofs) a 
statement of the necessary conditions for optimality for a class 
of problems that appear to be important as evidenced by recent 
research efforts. This class of problems is distinguished by 
the fact that the plant equations are most conveniently described 
by matrix differential equations. For such problems, it is im- 
portant to have a compact statement of the minimum principle 
so as to aid both intuition and mathematical manipulations; this 
provided the motivation for this study. 

In the remainder of this report the following topics are treated: 

1 • the relation of the matrix minimum principle 
to the ordinary minimum principle; 

2. a statement of the necessary conditions for 
optimality as provided by the matrix mini- 
mum principle; 

3. the solution of a very simple problem which 
involves the determination of the linear time - 
varying gains which optimize the response of 
a linear system with quadratic performance 
index. 

The most common form of the minimum principle pertains to the 
optimal control of systems described by vector differential equations 

x ciie J-Orrri 

x(t) = i[x(t),u(t),t] (1) 

(where x(t) is a column n-vector, u(t) is a column r-vector, and _f(* ) 

is a vector-valued function). These are the type of systems considered 

1 * 

by Pontryagin et al . and treated in most of the available books dealing 
with modern control theory. The description of plants by Eq. 1 is a 
very common one; however, there are problems in which the 
evolution -in-time of their variables is most naturally described by 
means of matrix differential equations. To make this more precise, 
consider a system whose state variables are x. ., with i = 1,2,... , n 


Superscripts refer to numbered items in the References. 
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and j = 1,2, . . . , m, and whose control variables are u q ^ , with 
a = 1, 2, . . . , r and (3 = 1,2, ...,q. In such problems, one may 
think of the " state matrix" ]£(t) whose elements are the state 
variables x^.(t) and of the "control matrix" U(t) whose elements 
are the control variables u a p(t), these are assumed to be related 
by the matrix differential equation 


X(t) = F[x(t), U(t), t] ( 2 ) 

where F[‘] is a matrix-valued function of its arguments. 

As an example of a system with this type of description consider 
a linear system 


M t) = _A ( t)_x ( t) + v_( t) 


(3) 


where v(t) is a white noise process with zero mean and covariance 

E {v(t)v' ( t)}= 6(t-r)C2(t) (4) 

If we denote by E(t) the covariance of the state vector x(t), i. e. , 

E(t) = E{x(t)x'(t)} (5) 

then it can be shown that I(t) satisfies the linear matrix differential 
equation 

L(t) = A(t)S(t)+S(t)A'(t)+Q(t) (6) 

which is in the form of Eq. 2. Indeed, there have been some 
applications of the matrix minimum principle to problems of 
filtering, control, and signal design (see Keferences 2 through 7). 
In these types of problems one is interested in minimizing a 
scalar -valued function of the covariance matrix E(t) and the 
"control variables" are some of the elements of the matrix 
A(t) and/or Q(t). 

If the system equations are naturally given by Eq. 2, it is 
easy to visualize an optimization problem. For example, consider 
a fixed -terminal time optimization problem with a cost functional 


T 

J(U) = K[X(T)1+ J L[X(t), U(t),t]dt 


t 

O 


(7) 
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where K[ • ] and L.[ - ] are scalar-valued functions of their argument. 
One may seek the optimal control -matrix U^(t), which may be con- 
strained by 


U*(t)eft (8) 

which minimizes the cost functional J. (U) . 

It should be clear that the tools are available to tackle this opti- 
mization problem. After all, one can decompose Eq. 2 into a set of 
first order equations 

x..(t) = f..[X(t), u(t),t] (9) 

and proceed with the application of the familiar minimum principle. 
What happens, however, is that one may get lost in a lot of equations 
and it may become almost impossible to determine any structure and 
properties of the solution. It is this possibility that "one may lose the 
forest from the trees” which has provided the motivation for dealing 
with problems involving the time -evolution of matrices by constructing 
a systematic notational approach. 

The first step towards this goal is to realize that the set of all, 
say, nxm real matrices forms a linear vector space with well- 
defined operations of addition and multiplication. Denote this vector 
space by S^, Then, it is possible to define an inner product in 
this space. Thus, if A and B are nxm matrices, i.e., Ae 
Be S^, their inner product is defined by the trace operation 

n m 

(A,B) = tr[ A B'] = £ £ a b (10) 

i=l j=l 

It is trivial to verify that Eq. 10 indeed defines an inner product. 

Using this notation one can form the Hamiltonian function for the opti- 
mization problem. First of all, note that if p_(t) is the costate 
variable associated with x..(t) then the Hamiltonian must take the 
form 

n n 

H = L[X(t), U(t), t] + (t)p (t) 

i-1 j=l 


(ID 
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Using Eq. 10, it follows that the Hamiltonian can be written as 


H = L[X(t), U(t), t] + tr [X(t)P'(t)] (12) 

where P(t) is the costate matrix associated with the state matrix 
X(t), in the sense that the costate variable p. ,(t) is the ijth element 
of P(t). 

8 

Using the notation of Athans and Falb, it is known that the costate 
variables satisfy the differential equations 




3H 


8x. .(t) 
U 


(13) 


This type of equation leads to the definition of the so-called gradient 
q 

matrix . Indeed it may be argued that the use of gradient matrices for 

purely manipulatory purposes is the key concept that makes the use 

of the matrix minimum principle suitable and straightforward. 

A gradient matrix is defined as follows: Suppose that f(X) is a 

scalar-valued function of the elements x. . of X. Then the gradient 
_ s 

matrix of f(X) is denoted by 


af(x) 


(14) 


and it is a matrix whose ijth element is simply given by 


dt(X) 


3(X) 


ij 


_ 8f &> 


(15) 


A brief table of some gradient matrices is given in Appendix A. 

Using the notion of the gradient matrix, it is readily seen that 
Eq. 13 can be written as 


P(t) = (16) 

since the Hamiltonian H is a scalar -valued function. 

Once this notation has been established, one can state all the known 
necessary conditions for optimality for vector -type problems to the 
equivalent statements for the matrix -type problems. In the following 
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section, the necessary conditions for optimality are stated for the 
fixed-time optimization problem with terminal cost. 



II. THE MATRIX MINIMUM PRINCIPLE (CONTINUOUS TIME) 

Consider a system with "state matrix" X(t)> "control matrix" 

U(t) e £2 described by the matrix differential equation 

X(t) = F[X(t),U(t),t] ;X(t 0 )=X Q ( 1? ) 

Consider the cost functional 

T 

J = K[X(T)1 + J L[X(t),U(t),t]dt ; T fixed (18) 
t 

o 

where K[ •] and L[ •] are sc alar -valued functions of their argument 
satisfying the usual differentiability conditions. 

Let P(t) denote the costate matrix. Define the scalar Hamiltonian 
function H by 


H[ X (t) , P (t) , t , U (t) ] = L[X(t),U(t),t] +tr[F(X(t), U(t), t)P'(t)] (19) 

If U*(t) is the optimal control, in the sense that it minimizes J, 
and if X*(t) is the corresponding state, then there exists a costate 
matrix P^ft) such that the following conditions hold 

(i) Canonical Equations : 


X*( 


4 -\ 

/ 


t tt * i-l 
^ l \ (ii) * * v / 9 ^ w » 


f20l 


p*(t) = 


8H 

8X(t) 


9 

8X*(t) 


= - 0x5%) L [2*(t).U*(t).t] 

tr[F(X*(t),U*(t),t)P*'(t)l 


( 21 ) 


(ii) Boundary Conditions : 

At the initial time 


X*(t ) = X 

— v o — o 


( 22 ) 
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At the terminal time (transver sality conditions) 


P*(T) = — K[X*(T)] 

9X ' (T) “ 

(iii) Minimization of the Hamiltonian : 

h[ x*(t) . P*(t) , t , U*(t)] < H[ X*(t)^ ! ' (t) , t , u] 


( 23 ) 


for every U e £2 and for each te[t Q , T] . (24) 

Note that if U(t) is unconstrained, then Eq. 24 im- 
plies the necessary condition 


9H 

9U(t) * 


0 


(25) 


i.e. , the gradient matrix of the Hamiltonian with respect 
to the control matrix U must vanish. 



III. THE MATRIX MINIMUM PRINCIPLE (DISCRETE TIME) 


There are problems for which the evolution of the pertinent variables 
is most naturally described by a set of matrix difference equations. 

For such problems, it is possible to extend the results of the ’Vector" 
discrete minimum principle^ ^ to obtain the equivalent form of the 
discrete matrix minimum principle. 

Consider the discrete optimization problem defined by a system of 
matrix difference equations 


-k+l " -k 


= £ k ®k^ k > 


0 . 1 , 


N-l 


(26) 


with U, ft 1 , X, e S for all k, and U. e S .. Consider the scalar 
— k J ’ — k nm ’ — k a(3 

cost functional 


N-l 

J = K(X n )+ L k (X k .U k ) (27) 

k=0 


It is assumed that F^_(-)> K(-), L^D satisfy the conditions re- 

quired by the discrete minimum principle. 

Define the Hamiltonian function 


H(X k .P k+1 ,U k > S (28) 

where P, is the costate matrix. 

— k 

Tf U*. k=0. 1 N-l is the optimal control and X* k=0, 1 N, 

— k' ... . — k 

is the optimal state, then the discrete matrix minimum principle 
states that there exists a costate matrix P*, k=0, 1, . . . ,N, such that 
the following relations hold 

(i) Canonical Equations : 


X 


-X* = 


-k+l -k 


9H 

8 £k + i 


= £ k (2 k . up 


T*\ 


* 


(29) 


_ "P 5 ! 5 _ _ 

-k+l -k " 9X k 


(3 0) 
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(ii) Boundary Conditions : 

At the initial M time f, (k=0) 

X* = X (31) 

— o — o 

At the terminal "time" (k=N) 



(iii) Minimization of the Hamiltonian: 

F or every U e !2 and each k=0, 1 , . . . , N-l 


H(XJ.P£ +1 .U£> <H<X*,P* +1 .U) (33) 

If the are unconstrained then Eq. 33 yields the neces- 

sary condition 


9H 

81 L 
— k 


= 0 


(34) 



IV. JUSTIFICATION OF THE MATRIX MINIMUM PRINCIPLE 


The extension of the vector minimum principle to the matrix case 

is straightforward. From a theoretical point of view it hinges on the 

existence of a mapping which relates the set of nxm real matrices 

to the set of (nm) -dimensional vectors. 

As before, let S denote the set of all real nXm matrices. Let 
’ nm 

R. . denote the (nm) -dimensional Euclidean vector space. Define 
(nm) 


a mapping ip from S nm into R 


(nm) 


ih : S -► R, x 
Y nm (nm) 


( 35 ) 


so that if X € S is the matrix 
— nm 


X 


then the image x e R^ nm ) 
dimensional column vector 


X 11 

X 1 2 

... x. 

lm 

X 21 

x 22 

* ' * x 2m 

1 * 4 

L X m 

• • • • 

X n2 

... x 

nm 

of X 

under 

the mapping 


( 36 ) 


x = 


X 


11 


X 


12 


Tm 


21 


22 


x 


2m 


[_ X nmj 


= 0(X) 


( 37 ) 
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lt is easy to verify that: 

1) */>(•) is a linear mapping 

2) ^(.) is one-to-one and onto, hence ip ^ exists 

3) ip(-) preserves the inner product because if X> ® nm an< ^ 

x e R^ nm ) so that x = ^(X),X = $(Y)» then the inner product 

(X, Y) in S is: 

n m 

(X, Y) = tr[XY>] = ^ ^ XjjYjj (38) 

i=l j=l 

while the inner product in ^ s: 


<s - x> = X I x ij y y 

i=l j=l 


so that <^(X), ip(Y)> = (X,Y) (40) 

Thus, the two spaces S nm and are algebraically and topo- 

logically equivalent. 

In the continuous time case, one starts from the matrix differential 
equation 

X = F(X,U,t) (41) 

Through the mapping rp this equation becomes 

* = i(x,U;t) (42) 

Similarly the integrand of the cost functional L«[X, U» *1 changed 

into L[x, U, t] . Then, by the ordinary vector minimum principle, 
there is a costate vector £€ R^ nrn ) associated with xe Let 
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P 11 


P 


12 


£ 


p 21 

p 22 


(43) 


p 2m 

• • ft 

• • « 
p 

r nm 


Then the Hamiltonian function in the vector case is 


H = L(x ,U,t) + <x,£> (44) 

Since exists one can find a unique co state matrix Pe 

P = f 1 (£) (45) 

so that the Hamiltonian H can be written as 

H = L(X,U,t) + (X.P) (46) 

in the matrix case. Thus, the fact that $ preserves the inner product 
(involved in the definition of the Hamiltonian) coupled with the specific 
definition of the gradient matrices yields the matrix minimum principle 
in the continuous -time case. 

Caution ; If X is constrained to be symmetric, then the mapping 
ip(') is not invertible. In this case, the definitions of the gradient 
matrices and the formulae of Appendix A are not valid so that the state- 
ments in Sections 2 and 3 must be modified in order to obtain the cor- 
rect answers . 



V. APPLICATION TO A LINEAR CONTROL PROBLEM 


In this section the matrix minimum principle is used to determine 
the solution to the simple optimal linear regulator problem. Consider 
a linear time -varying system with state vector x (t) and control vector 
u (t) related by the vector differential equation 

x (t) = A(t)x(t) + B(t) u(t) (47) 

where A(t) is an nxn matrix and _B(t) an nxr matrix. Consider 
the quadratic cost functional 

T 

J = J [x'(t)Q(t)x(t) + u'(t)R(t)u(t)]dt (48) 

t 

o 

where Q(t) and R(t) are symmetric positive definite matrices. The 
standard optimization problem is to find the control u(t), t <_ t < T, 
so as to minimize the cost functional J. 

Instead of dealing with this standard problem, consider the fol- 
lowing variation. Suppose that one imposes the constraint that the 
control u(t) be generated by using a linear time -varying feedback law 
of the form 


u(t) = -G(t)x(t) (49) 

where G(t) is an rxn time-varying "gain" matrix (the elements of 
G(t) specify the time -varying feedback gains which multiply the ap- 
propriate state variables). In this case, the system satisfies the 
closed-loop equation 


X(t) = [A(t)-B(t)G(t)]x(t) 
and the cost functional J reduces to 


J 


T 

J x 1 (t)[ Q(t)+G' (t)R(t)G(t)] x (t) dt 


t 

o 


(50) 


(51) 
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To complete the transformation of the problem into the framework 
required by the matrix minimum principle, define the nxm "state 
matrix" X(t) as the outer vector product of the state vector x(t) 
with itself, i.e.. 



X(t) = x(t)x'(t) 

(52) 

Noting that 

x' (t)x(t) = tr[ X(t)] 

(53) 


x'(t)F(t)x(t) = tr[ F(t)X(t)] = tr[X(t)F(t)l 

(54) 


it follows from Eqs, 52 and 50 that 


X(t) = *(t)x'(t)+x(t)*'(t) = [ A(t)-B(t)G(t)]x(t)x'(t)+x(t)x'(t)[ A(t)-B(t)G(t)] ' 

(55) 

so that the state matrix X(t ) satisfies the linear matrix differential 
equation 

X(t) = [ A(t)-B(t)G(t)lX(t) + XW (56) 

with the initial condition 

X(t Q ) = x( t 0 )x , (t o ) (57) 

The cost functional J reduces to 

T 

J - f tr[ (Q(t)+G'(t)R(t)G(t))X(t)] dt (58) 

t 

o 

The system (56) and the cost functional (58) are in the form re- 
quired to use the matrix minimum principle* So let P(t) be the 
nxn costate matrix associated with X(t). The Hamiltonian function 
H for this problem is* 

H = tr[QX] + tr[ G'RGX] + trf AXP 'l -trf B GXP 'I +tr[XA'P'] -trfXG'B'P'] 

(59) 


The time dependence is suppressed for simplicity 
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The canonical equations yield (using the gradient matrix formulae of Ap- 
pendix A) 

X = |p = [ A-BG]X+ X[ A-BG] ’ (60) 

P = - ~ = "9 - G'RG- [ A - BG] 'P-P[ A-BG] (61) 

The boundary conditions are 

X(t Q ) =x(t Q ) x'(t Q ) ; P(T) = 0 (62) 

Since G is unconstrained, it is necessary that 

0 = = RGX' + RGX - B'PX' - B'P'X (63) 

Note that both X(t) an ^ P(t) are symmetric. To see this, note that 
the solution of Eq. 60 is; 

X(t) = $(t,t o )X(t 0 )$'(t,t o ) (64) 

where $(t,t Q ) is the transition matrix of [ A(t) -B(t)G(t)] . The sym- 
metry of X(t) follows from Eq. 64 and the symmetry of X^ 0 )* -A- 

similar argument can be used to establish the symmetry of P(t). These 
symmetry properties and Eq. 63 yield 

[ R(t)G(t) -B' (t)P(t)] X(t) = 0 (65) 

If this equation is to hold for all X(t), then one deduces 

G(t) = R _1 (t)B'{t)P(t) (66) 

To completely specify the gain matrix G(t) one must determine 
the costate matrix P(t) . By substituting Eq. 66 into Eq. 61 one finds 
that the costate matrix P(t) is the solution of the familiar Riccati 
matrix differential equation 


* 


This is the same argument that one uses in the vector case to obtain 
the feedback solution; see Ref. 8, p. 761. 
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P(t) = -P(t)A(t) - A'(t)P(t) + P(t)B(t)R -1 (t)B'(t)P(t)-Q(t) (67) 

with the boundary condition 

P(T) = 0 (68) 

It should be clear that the necessary conditions provided by the 
matrix minimum principle yield the same answer that one would obtain 
in the vector formulation. It is, of course, well known that the answer 
is indeed the unique optimal one. 

The fact that the costate matrix P(t) is the solution of the Ric- 
cati equation sheds some light in its physical interpretation. If we 
view, as required by the Hamilton-Jacobi -Bellman theory, the costate 
matrix as the gradient matrix of the cost with respect to the state, i.e. , 

= ax(tj (69) 

it is evident that the Riccati equation defines the evolution of the partial 

derivatives 9j/9x_(t) for te[t q , T] . This conclusion cannot be 
reached as readily in the vector formulation of the problem. 



VI. CONCLUSIONS 


It has been shown that systems described by matrix differential 
and difference equations can be optimized by the matrix version of the 
minimum principle of Pontryagin. The definition of the gradient 
matrix of a scalar -valued function of a matrix facilitates the manipu- 
lation of the necessary conditions for optimality as illustrated by the 
problem of optimizing the gains of a linear system. 
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APPENDIX A 


A PARTIAL LIST OF GRADIENT MATRICES 

The formulae appearing below have been calculated in the un- 

o 

published report by Athans and Schweppe. Some of them have also 
been calculated by Kleinman using a different approach (Appendix F of 
Reference 5). The interested reader should consult these reports for 
details. The results are stated in this appendix for the sake of refer- 
ence; the calculations involved are straightforward but lengthy. 

In the formulae below X is an nxm matrix. The reader is 
cautioned that the formulae are not valid if the elements x. . of X are 


not independent. 


^tr[X] = I 

(A. 1) 

j^tr[ AX] = A' 

(A. 2) 

g^tr[AX'] = A 

(A. 3) 

^tr[AXB] = A'B' 

(A. 4) 

■— tr[ A X'B] = BA 

(A. 5) 

tr[ AX] = A 

(A. 6) 

tr[ AX'] = A' 

(A. 7) 

tr[ AXB] = BA 

(A. 8) 

9 ^, tr[ AX'B] =A'B' 

(A. 9) 

^tr[XX] □ 2 X ' 

(A. 10) 

^tr[XX'] = 2X 

(A. 11) 

^tr[X n ] =n(X n_1 ) 1 

(A. 12) 


ij 

n-1 

0^tr[ AX n ]=(^T x\A2f"' L ' 1 )' (A. 13) 

i=0 

g^tr[ AXBX] = A'X B'+ B'X' A' (A. 14) 
^tr[ AXBX 1 ] = A'XB'+ AXB ( A . 1 5) 
^tr[e— ] = e- (A. 16) 

^tr[ X - 1 ] = -(X -1 X _1 ) ' = " (X ~ Z ) ’ ( A . 1 7) 
^tr [ AX " 1 B] = -(X _1 B AX _1 ) * ( A . 1 8) 

^det[x] = (det[X])(X“V (A. 19) 
^log det [X]=(X"V (A. 20) 

g|^det[ AXB]=(det[ AXB])(X"V (A. 21) 
^et[X]^et[x]=(detlxl)(X' q )' (A. 22) 
^det[X n ] =n(det[X]) n (X -1 )' (A. 23) 
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